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ABSTRACT 


Schools across the United States suffer from low on-time 
graduation rates. Targeted interventions help at-risk stu- 
dents meet graduation requirements in a timely manner, but 
identifying these students takes time and practice, as warn- 
ing signs are often context-specific and reflected in a combi- 
nation of attendance, social, and academic signals scattered 
across data sources. Extremely high caseloads for counselors 
compound the problem. At Infinite Campus, a large student 
information system provider, we modeled statistical rela- 
tionships between student educational records and enroll- 
ment outcomes, using de-identified records and in-system 
analysis to guarantee student data privacy. The resulting 
risk scores are highly predictive, context-sensitive, nation- 
ally available, integrated into the existing student informa- 
tion system, and updated daily. 


1. INTRODUCTION 


Approximately 15% of American students do not graduate 
high school on time [16]. States and districts frequently 
employ interventions designed to improve educational out- 
comes, including reducing dropout rates. A key role of 
school counselors is to direct the application of these inter- 
ventions to the students who need them most. Counselors 
first need to identify these students but are faced with infor- 
mation overload. Each student’s data is distributed across 
a student information system (SIS) and often other systems 
or people, making it difficult to synthesize into an accurate, 
comprehensive portrait of a student’s risk. Compounding 
the problem are extremely large counselor caseloads—the 
national average is 430 students per counselor—with higher 
numbers typical in schools serving children with other struc- 
tural disadvantages [11]. 


Early warning systems function as “automated attention” for 
overworked counselors by automatically identifying students 
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who might benefit from additional institutional resources. 
They automate the more tedious data analysis and summa- 
rization tasks so that counselors can focus on what humans 
do best: building relationships. 


1.1 Alternate approaches 

An effective dropout prevention system requires developing 
people, processes, and technology [7], identifying valid pre- 
dictors, managing data and reports, assigning interventions, 
and monitoring student progress [10, 15]. In this paper, we 
focus just on the technology that identifies risk factors and 
estimates student dropout risk, which can then be embedded 
in a larger dropout prevention system. 


Quantitative and qualitative determination of school dropout 
risk factors is a decades-old area of research [19], though 
the mid-2000s were a particularly important inflection point. 
High-profile studies of dropouts in the Chicago [3] and Philadel- 
phia [17] urban districts led to the development of statisti- 
cal methods for determining risk factors and their incor- 
poration into early warning systems. Several organizations, 
often working together, have been instrumental in encourag- 
ing American schools to adopt research-based best practices 
[10], including the U.S. Department of Education’s Regional 
Education Laboratories [18], the Consortium on School Re- 
search [2] and NORC [7] at the University of Chicago, the 
American Institutes for Research [8], and the Everyone Grad- 
uates Center at Johns Hopkins University [9]. Most states 
now make an early warning system available to their school 
districts [6]. 


Until the mid-2010s, all widely-used dropout early warning 
systems used threshold-based models, characterized by a few 
easily comprehensible predictors with associated risk thresh- 
olds (e.g., failing at least one course or being absent at least 
20 days). The simplicity and auditability of these models, 
and the associated ease of implementation using common 
software and spreadsheet skills, is their key advantage over 
machine-learned systems. Students are measured on each 
predictor and flagged as “on-track” or “off-track” based on 
which side of a preset threshold their data point falls. Staff 
can intervene with students who have the most “off-track” 
risk flags, or whose risk areas correlate with particular in- 
tervention domains. A particularly influential approach is 
Balfanz’s “ABC” taxonomy, in which students are measured 
on attendance, behavior, and course performance metrics 
[4], optionally with different thresholds for different student 
subpopulations [12]. While some educational institutions 
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implemented an ABC-style system themselves [10], others 
used spreadsheets or data tools made available by organi- 
zations like American Institutes for Research [8]. An active 
area of research involves determining which predictors and 
thresholds are appropriate for each school, or whether single 
thresholds are appropriate at all. 


To overcome the limitations inherent in threshold-based sys- 
tems, we and other organizations created machine-learned 
dropout risk identification systems in the mid-2010s. A 
number of researchers describe their systems in the aca- 
demic literature (e.g., [1]) or industry white papers (e.g., 
[20]). Several organizations serve machine-learned dropout 
predictions at the scale of hundreds of thousands of students 
in many school districts. The Wisconsin Department of Pub- 
lic Instruction’s Dropout Early Warning System uses data 
reported to the state every few months by the districts’ SISs 
to build machine-learned models. The system produces two 
predictions per year and is available for students in grades 
6-9 [14]. Mazin Education (through BrightBytes) and Hoo- 
nuit both sell machine-learned early warning and interven- 
tion monitoring systems for all grade levels. 


Machine-learned systems have two major advantages over 
threshold-based models. First, the additional model com- 
plexity affords more accurate predictions and allows system 
designers to infer which risk factors are predictive in the 
presence of other factors or for different populations. Sec- 
ond, the variety of model architectures allows for more than 
just inferring overall risk. Designers can choose, for instance, 
to model time until dropout so that staff can intervene ac- 
cording to acuteness of risk, or to model uncertainty. 


1.2 Our contribution 

Two key obstacles prevent machine-learned early warning 
systems from being deployed nationwide. First, the pre- 
dictive quality of these models is chiefly a function of data 
availability, as models must be trained on a large dataset— 
including a variety of educational contexts and outcomes— 
to ensure they perform well for students they haven’t seen 
before. Models built on a single district’s or even state’s 
data may not generalize well to other populations. How- 
ever, building a model on multiple states’ data requires the 
data to be standardized, and without a common SIS to en- 
force uniformity, manipulating data into a common format 
is costly and time consuming. 


Second, predictions must be surfaced to educators in a fre- 
quent and easy-to-use manner, which means the most suc- 
cessful systems will be closest to existing daily workflows. In 
some existing systems, staff members must log into a sepa- 
rate software program to access risk scores. In others, there 
are months between score updates. Timely prediction is im- 
portant; the sooner a school is aware of a student at risk, 
the more time it has to intervene. 


Infinite Campus provides a large student information system 
and has made significant investment in education data local- 
ization and standardization, reporting and warehousing, and 
user workflows for American K-12 education. Our role in 
the industry positions us to address the key remaining gaps 
in early warning systems using centralized data warehouses, 
standardized data, and placement of the early warning ap- 


plication into the existing SIS. 


Contributions 

Available nationally in 32 states yet contextual to each 
child’s educational environment 

Useful predictions: highly predictive, daily updates, 


four risk category scores, with consistent of predictive 
quality between protected student groups 

Integrated into the student information system with no 
imports, exports or synchronization necessary 


In the following section we describe our implementation of 
a dropout early warning system on more than 6 million 
student-years of educational records across 32 states. Our 
overall dropout risk score has excellent predictive quality 
with an AUC of 0.941 (see table 1 for additional quality met- 
rics), and includes additional machine-learned scores that 
help counselors understand the source of a student’s risk to 
guide which interventions may be appropriate. Risk scores, 
delivered automatically and updated daily, are integrated 
into our existing SIS and available to counselors as an en- 
hancement to their existing workflows. 


2. IMPLEMENTATION 


System design. Student data is stored in a number of rela- 
tional SQL Server databases for school staff to create, read, 
update, and delete educational records. These databases ex- 
ist in Infinite Campus’s fully owned and operated Tier 4 data 
centers that fulfill security requirements of the U.S. Depart- 
ment of the Interior. Student records exist in a variety of 
data structures and are recorded with varied time granular- 
ity for 45 states and the U.S. federal government, a portion of 
which we use for early warning. Because our model architec- 
ture requires a common data structure, we aggregate student 
records into a fixed format with one row per student-year, 
where ‘year’ corresponds to an academic year. Aggregated 
data is periodically transferred to a central repository in the 
same data center. Data for past students whose educational 
outcomes are known (e.g., graduation or dropout) is then 
used to build a machine-learned model relating summarized 
educational records to student enrollment outcomes. The 
model, along with summarized properties of each district, 
school, and geographic area present in the dataset, is de- 
ployed behind an API. Each day, records for currently en- 
rolled students are aggregated, de-identified, and sent to the 
API, which returns risk scores for each student. Figure 1 
illustrates this architecture. The returned risk scores and 
a score history is made available to counselors via the SIS 
user interface. By integrating and automating the process 
of generating and updating risk scores into the SIS, we re- 
lieve dropout prevention teams from the burden of collect- 
ing, storing, and analyzing the source data themselves. 


What we predict. Each student-year of aggregated educa- 
tional records is tagged with one of three labels: “needs early 
warning”, “does not need early warning”, or “ignore for early 
warning”. A student-year is labeled with “needs early warn- 
ing” if the student’s records include known undesirable out- 
comes during the year in question or future years. For ex- 
ample, records for an 8th grader in 2014 would be labeled 
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Figure 1: Schematic of architecture describing data flow for model training and prediction. Model train- 
ing is performed infrequently, while risk scores are recalculated daily to incorporate new information. All 
data remains in the Infinite Campus data center and is not transferred to third-party servers. Identifying 


information is removed before transfer between systems. 


“needs early warning” if they dropped out in 2016. We define 
undesirable outcomes based on enrollment end status codes 
identified by states as indicating school dropout, and expand 
the definition to include unsatisfactory academic progress 
(i-e., retention or demotion) and expulsion or other removal. 
A student-year is labeled with “does not need early warning” 
if the student had no undesirable enrollment outcomes dur- 
ing the year in question or future years, and we can confirm 
that the student persisted to graduation. If we do not know 
whether the student persisted to graduation—the student 
transferred outside of our system or is still enrolled—then 
the student’s data is censored and we lack ground truth la- 
bels for it. Each of these students’ years are labeled with “ig- 
nore for early warning” and excluded from training and eval- 
uation. To translate state-specific enrollment status codes 
to outcome categories, codes were mapped by three indepen- 
dent raters, then differences were reconciled and validated 
by comparing resulting outcome rates across districts. Data 
from a school district are removed from training if that dis- 
trict has abnormally high “needs early warning” or “ignore 
for early warning” rates, as these conditions may indicate un- 
derlying inconsistencies in record-keeping that warrant fur- 
ther investigation. In addition, student-years are removed 
from training if the student’s cohort is not scheduled to have 
graduated yet, in order to remove label bias in earlier grades. 


Our data collection and labeling process produces approxi- 
mately 6.4 million rows of labeled data. We use 45% of rows 
for model training, 5% for model validation (to determine 
training stopping points), and 50% for final quality evalu- 
ation, split by student. Roughly 16% of training rows are 
labeled as “needs early warning”. 


Predictors. Our training set is produced by collecting and 
summarizing educational information from the core SIS data- 
base. This summarized information relates to attendance, 
academic performance, behavior, household and enrollment 


stability, and other items. We chose predictors that are con- 
sistent across states and districts and that are supported by 
the dropout prevention literature (e.g., [19]). Where data 
is localized, we employ experts who communicate directly 
with stakeholders in districts and states to ensure we under- 
stand the unique characteristics of local use and law. At- 
tendance information includes the proportion of class time 
a student was actually present, as well as absences grouped 
by type of excuse. Academic performance information in- 
cludes the proportion of course grades attributed to each 
letter grade, overall high school GPA, and the proportion of 
attempted credits successfully earned. Behavior information 
includes the number of behavior infractions and resolutions, 
as well as whether weapons, drugs, or harassment were in- 
volved. Household and enrollment stability information in- 
cludes the presence of past undesirable enrollment outcomes 
and how often the student changes home addresses, schools, 
or districts in the middle of school years. Finally, we include 
contextual information such as age and grade level. In to- 
tal, we have approximately 70 distinct predictors per year, 
and each student-year row includes the current and previous 
year’s data. 


One core design goal for our system was to have a nation- 
wide statistical model that is sensitive to local and contex- 
tual factors. We achieved context-specific performance with 
two types of feature engineering: including subpopulation 
aggregation features and calculating interactions among a 
student’s personal features as well as the subpopulation ag- 
gregates that apply to them. For example, a student’s at- 
tendance or academic data relative to their peers in a given 
group may carry information about risk. To allow for this 
possibility, we calculated two types of summary statistics 
for each school, district, and ZIP code in our dataset. For 
numeric predictors, we calculated the mean value per group. 
For categorical predictors, we calculated the proportion-per- 
group of each category. These group-level contextual fea- 
tures allow us to capture signals about students’ environ- 
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ments that are more informative than simple group-membership 


variables. The group-level information about each student- 
year’s specific district, school, and ZIP code is joined on each 
student-year row for model building and prediction. 


By including a wide range of features characterizing individ- 
ual students and their educational environments, we allow 
the machine learning model to determine the significance of 
each feature and the relationship between features as they 
relate to risk. The wide range of potentially predictive fea- 
tures is simply not available to most non-SIS vendors. By 
using a relatively complex feature set and modeling architec- 
ture, we are able to capture complex contextual relationships 
between students and their environments. 


Explanatory scores. While machine learning affords high 
predictive accuracy and the ability to capture complex re- 
lationships between predictive features, this comes at the 
cost of reduced model interpretability. In early conversa- 
tions with customers, we were frequently asked why a stu- 
dent received a given risk score. To answer this question, we 
supplement our overall risk score with two types of explana- 
tory ‘category’ scores that provide insight into which parts 
of a student’s record are contributing to their overall risk. 


The first set of explanatory scores is based on Balfanz’s 
“ABC” categories [10]—attendance, behavior, and course 
performance—and an additional “stability” score including 
measures of household and enrollment stability. Each of the 
four scores is produced by a separate model trained only 
on predictors from its respective category. By partitioning 
the predictors according to category, we in turn are able 
to disentangle the impact each category has on the over- 
all risk score. The predictive quality of category scores is 
necessarily lower than the overall risk score, because the 
category-specific models use a strict subset of the overall 
model’s predictors. However, these scores indicate whether 
each category of a student’s predictors, when taken by it- 
self, is characteristic of a student with undesirable future 
enrollment outcomes. 


In addition to scores built on subsets of predictors from 
each category, we also build scores for each category using a 
“counterfactual” approach. That is, if a student’s records im- 
proved in a certain area (but the rest of the student’s records 
stay the same), how would their risk change? To answer this 
question, we replace the values for the “actionable” predic- 
tors in a category with values corresponding to exemplary 
performance. The resulting data represents an attainable 
ideal for each student—if he or she attended every class, 
earned perfect scores on every assignment, or never behaved 
inappropriately. This data is used to produce a counterfac- 
tual risk score for each of the four categories, which when 
subtracted from the student’s actual overall risk score indi- 
cates the potential “room for improvement” in each category; 
these are our final four explanatory scores. 


While picking values corresponding to exemplary perfor- 
mance appears intuitive (e.g., 4.0 GPA, 100% attendance), 
using them to artificially modify student data has the po- 
tential to push the resulting data points outside the space 
of training examples, leading to unpredictable model behav- 


ior. Preliminary analysis found this to be a problem for some 
“obvious” exemplary values, leading us to select values ex- 
perimentally instead. For each feature, we used a statistical 
model to find the optimal bin (range of values) correspond- 
ing to the lowest predicted risk, which we subsequently val- 
idated by checking that the proportion of actual dropouts 
was lowest for this bin. We chose a reasonable value from 
each optimal bin to represent exemplary performance. 


2.1 Modeling technique 

The system described here must operate at scale within an 
industry setting and be robust to messy and missing data. 
To achieve this, we use the xgboost package [5] for model- 
ing, which constructs a series of simple decision trees. Unlike 
logistic regression or neural networks, xgboost is robust to 
the presence of outliers and appropriately handles missing 
values. The decision-tree structure of model components 
provides an integrated way to capture contextual relation- 
ships between individual predictors and group-level aggre- 
gates. xgboost supports parallelization of model training, 
so training scales well on enterprise server hardware. We use 
the ‘binary:logistic’ training objective, so that xgboost mod- 
els produce the probability of a student-year being labeled 
as “needs early warning”. We use the area under the receiver 
operating characteristic curve (AUC) as the xgboost evalu- 
ation metric. AUC measures the quality of sorting produced 
by the model, with a high value for AUC indicating that the 
model is correctly assigning higher probabilities to student- 
years labeled as “needs early warning” than to those labeled 
as “does not need early warning”. 


Our modeling strategy ensures robustness to noise in two 
ways. First, we heavily regularize our xgboost models by 
using a small tree depth and relatively few training rounds, 
reducing over-fitting and making the model more robust to 
small changes in student data (both during training and dur- 
ing prediction). Regularization makes it possible to provide 
high-quality predictions for unseen data, such as a student 
in the evaluation dataset or a new customer whose data was 
not included in model training. Second, for each score type 
we train an ensemble of 11 to 25 xgboost models, and use 
the median prediction of all models. This technique further 
reduces variability between model deployments. 


Predictions. Predictions are refreshed daily by aggregating 
educational records of currently enrolled students and send- 
ing those aggregates to our API, as illustrated in Figure 1, 
which provides GRAD scores and category scores back to 
the SIS. This technique allows us to provide score updates 
more frequently than competitors, and eliminates the re- 
quirement for school districts to transfer or analyze data on 
their own. The SIS then displays risk scores to staff mem- 
bers that have been given access to the early warning tool 
by district administrators. 


As described above, we train the model using aggregations 
from past entire student-years. However, daily predictions 
are made for currently enrolled students, whose current year 
records contain only a partial year of data. We used sev- 
eral methods to mitigate the mismatch with our training 
dataset. First, in addition to aggregates summarizing a 
student’s data from the current academic year, we also in- 
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clude aggregates from the student’s previous year as predic- 
tors. This allows the model to observe student data for an 
already-completed year and it affords analysis of year-over- 
year changes. Second, some data is in the form of event 
counts that accumulate throughout the year, such as the 
number of missed periods or the number of behavior resolu- 
tions. This data is converted into a rate, such as missed peri- 
ods per instructional day. Rates are then directly compara- 
ble at all points in the academic year. Since rate-converted 
values are sensitive to small changes at the beginning of the 
school year, we instead use an ‘estimated rate’ calculated as 
a weighted combination of the previous and current year’s 
rates. After about four weeks of school, the previous and 
current year’s rates are equally weighted, with the current 
year’s rate weighted more heavily after each additional day. 


We convert the probability output from xgboost into a user- 
facing “GRAD score” that ranges from 50 to 150, where 50 
indicates high likelihood of undesirable enrollment outcomes 
in the future and 150 indicates high likelihood of persistence 
to graduation. We also considered that counselors may have 
a greater need to distinguish between students with low and 
moderate risk rather than between students with high and 
very high risk. That is, a 0.1 change in dropout probability 
from 0.05 to 0.15 is more important than a change in prob- 
ability from 0.75 to 0.85. We therefore transform the raw 
probabilities to ‘spread apart’ students at the low end of the 
probability range (low risk), while compressing probabilities 
at the high end of the range (high risk). 


2.2 Model evaluation 

We evaluated our overall and subscore models on an evalua- 
tion set containing approximately 3.2M student-years (50% 
of the total dataset) that were not used for model building or 
validation during training. Results are listed in Table 1 and 
represent, as far as we are aware, the highest predictive qual- 
ity in the industry. In Table 2, we also list evaluation results 
for our overall model by protected subpopulation [13]: sex, 
race/ethnicity, grade level, and free/reduced meal eligibility 
(a proxy for socioeconomic status). 


Because counselors do not see predicted probabilities, but 
rather ordered GRAD scores, we chose the area under the 
receiver operating characteristic curve (AUC) metric that 
measures whether students’ predictions are ordered in terms 
of actual risk. The AUC effective range is 0.0 (perfectly 
inversely sorted) to 1.0 (perfectly sorted), with 0.5 indicating 
random predictions. 


Futhermore, because counselors will give additional insti- 
tutional resources to the few percent of students predicted 
most at-risk, we chose precision and recall metrics that mea- 
sure whether how well that most at-risk prediction category 
actually contains at-risk students. We evaluated precision at 
10% (P@10) and recall at 10% (R@10) following the litera- 
ture [1]. A key limitation of precision@k and recall@k occurs 
when & is less than the population’s condition-positive rate, 
and therefore the effective range of those metrics is less than 
1.0. To correct for this limitation, we also measured preci- 
sion=recall at 16% (PR@16) because the condition-positive 
(“needs early warning”) rate of the training set is 0.158. For 
subpopulation evaluation, precision and recall at baseline 
(PR@b) is based on that subpopulation’s own condition- 


PRGIG 


GRAD Score 


Academics 


Attendance 
Behavior 
Stability 


Table 1: Risk score quality evaluation 


Female 0.130 | 0.935 | 0.770 | 0.597 | 0.683 
Hispanic : f ; : : 
Asian 
AIAN 
NHPI 
2+ races 
White 
Black 
Not stated 
grade 
" grade 
" grade 
" grade 
10 grade 
11° grade 
12° grade 
NSLP: Free 
NSLP: Reduced 
NSLP: Paid 
NSLP: N/A 


Table 2: Overall model risk score quality for subpop- 
ulations. ‘+ rate’ refers to the baseline ‘condition- 
positive’ negative enrollment outcome rate for 
that subpopulation’s current or future enrollments. 
NSLP refers to the National School Lunch Program. 
AIAN refers to American Indian and Alaska Na- 
tive. NHPI refers to Native Hawaiian and Pacific 
Islander. * means that the ‘+ rate’ value is less 
than 0.1 and therefore the effective maximum range 
of P@10 and R@10 is less than 1.0 for that subpop- 
ulation. 


positive (“needs early warning”) rate. The effective range of 
PR@16 and PRQb is 0.0 (completely incorrect) to 1.0 (com- 
pletely correct), with the random prediction rate equivalent 
to the baseline rate for that population. 


3. LIMITATIONS AND EXTENSIONS 


The use of year-level aggregates in model training erases 
temporal relationships between individual event records, mak- 
ing the system relatively blind to patterns of individual events 
within an academic year. To address this limitation, we are 
exploring alternate modeling strategies capable of ingesting 
event-based data streams that are both more granular and of 
non-uniform length. A second limitation is our model’s focus 
on grades 6-12. Interventions are most successful when they 
are applied early [21]. The ability to provide meaningful risk 
indicators for younger students could significantly improve 
outcomes by helping counselors target interventions toward 
the students who need them most, at the point they can ben- 


Proceedings of The 12th International Conference on Educational Data Mining (EDM 2019) 730 


efit from them most. To do this, we must overcome several https: //ccrscenter.org/ccrs-landscape/ 
data-consistency related obstacles, the most pressing being state-profile/new-state-profile-comparison. 
a lack of long-term datasets that are consistent in the type [7] L. Cordeiro, R. Blank, L. Hansen, D. Leeds, and 
of information collected over time and the quality /reliability L. Selfa. Considering the best early warning system 
of the collection. (EWS) model to fit your needs, May 2016. 
[8] A.-M. Faria, N. Sorensen, J. Heppen, J. Bowdon, 
S. Taylor, R. Eisner, and 8. Foster. Getting students 
on track for graduation: Impacts of the early warning 


Finally, although target labels were created using inter-rater 
reliability methods, research on state policies, and student 


outcome data, the labels have not been verified by repre- intervention and monitoring system after one year. 
sentatives from each school district who could personally https: //eric.ed. gov/?id=ED573814, 2017. 


attest to the accuracy of a given student’s outcome. We in- [9] J. H. Fox, E. S. Ingram, and J. L. Depaoli. For all 
tend to make the target calculation available to schools and kids: How Kentucky is closing the high school 

to implement a system for users to provide feedback on our graduation gap for low-income students. 
product’s predictive accuracy to allow us to verify our labels https: //eric.ed.gov/?id=ED572766, 2016. 


and to continue to improve the quality of predictions. [10] S$. Frazelle and A. Nagel. A practitioner’s guide to 
implementing early warning systems. Regional 

4. CONCLUSION Educational Laboratory at Education Northwest, 2015. 

We built a decision support system that provides high-quality, [11] D. J. Gagnon and M. J. Mattingly. Most US school 

context-sensitive risk predictions and is integrated into an districts have low access to school counselors: Poor, 

SIS that thousands of counselors already use in their work- diverse, and city school districts exhibit particularly 

flows. In doing so, we offer daily risk assessments to mil- high student-to-counselor ratios. 

lions of currently enrolled middle and high school students https://scholars.unh.edu/cgi/viewcontent.cgi? 

across the country. By automatically identifying students article=1285&context=carsey, 2016. 

who may benefit from additional institutional resources in [12] Key data for an “early warning system” with on- and 

the service of timely graduation, we fulfill a key component 

of the dropout prevention process. 


off-track indicators that become the basis for tiered 
interventions. http: //guidebook.americaspromise. 
org/tools-directory. 
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